User interface and algorithm to handle “unknown” data in card-sorting exercise and cluster analysis

ABSTRACT

Card-sorting exercises are used to understand how users would intuitively group or sort information topics, in order to better design an instrument that provides these topics, such as a website. When a user is not familiar with some of the topics, they are allowed to leave these items unsorted, in order that wild guesses do not skew the results. The algorithm verifies that the unsorted items are unfamiliar, then tracks instances of unsorted items so that these responses are mathematically removed from the calculations.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates generally to a method of gathering andcorrelating user input regarding relationships between topics, thisinformation being useful in designing web sites, program interfaces, andmany other information design applications. Specifically, the presentinvention provides a method and algorithm for handling a lack of userinput in portions of the data gathering process where the user isunfamiliar with some of the items presented.

2. Description of Related Art

Card sorting is a technique used by the builders of web sites toorganize the information on the site and to decide how to label thecategories for ease of use. The technique works by gathering data from anumber of users regarding their perception of relationships betweentopics. The strength of the perceived relationships can then drive thedesign of the site.

In a manual version of card sorting, a user is given a set of indexcards containing likely topics for the site, one topic per card. Theuser then sorts the cards into groups according to his perception ofwhich topics belong together. Note that in this exercise, there is noright or wrong way to sort items. This is a subjective exercise thatseeks to discover perceptions. Therefore, different users will have atendency to group items differently, especially as the ideas theyrepresent become the more complex.

The input from a number of users can then be correlated in a matrixaccording to how closely users group each set of two cards together, amethodology known as cluster analysis. Manual correlation and analysis,however, can be tedious.

EZSort is a software package created by IBM, Inc., which handles thecard sorting process and analysis. EZSort has two parts—USort andEZCalc. USort handles the card sorting exercise for all participants;EZCalc performs cluster analyses on the accumulated data and generatestree diagrams that represent the hierarchical relationships.

FIG. 1A shows a computer screen containing a typical card-sortingexercise handled by USort. In this figure, the “cards” to be sorted arepresented on the left side of the screen (the source); the right side ofthe screen (the target) is where a user sorts the cards into groupsseparated by horizontal lines, using drag and drop operations. Noticethat the cards to be sorted include a wide variety of topics includinghardware, software, languages, operating systems, interfaces betweenusers, interfaces between computers, etc. A user's background andexperience will tend to affect the way that he would perceive items asbelonging together.

When the user is satisfied with the groupings, clicking on the rightarrow (110) causes the program to move to the next step, seen in FIG.1B. On this second screen, the user is allowed to designate further,higher-level groupings, if these are deemed desirable. The previouslyformed groups are presented. The groups can be rearranged, and largergroupings formed by making the lines between high-level groups intodouble lines. In a third step, which is not shown, the user is allowedto name the categories into which he has grouped items. Once theexercise is complete, the users information is saved to a file for laterprocessing.

When all card-sorting exercises have been done, the data goes to EZCalcfor analysis. A raw score matrix is created for each participant,according to the following. If two items are not grouped together by theparticipant, a value of 0 is assigned. If the two items are groupedtogether in a high-level grouping, but not in the low-level grouping, avalue of 1 is assigned. If the two items are grouped together in boththe high-level and low-level groupings, a value of 2 is assigned. Thus,each possible pairing of items receives a score of 0, 1, or 2. Next, theraw scores for each pair of items are summed together for all of theparticipants, forming a total raw score matrix. The values in thismatrix are normalized into a similarity matrix by dividing each score by2·n, where n is the number of participants. Each element in thesimilarity matrix now has a score of 0 to 1. Items in the similaritymatrix are converted into a distance matrix, using the formulaD(x,y)=1−S(x,y)where D(x,y) is an element in the distance matrix for card pair x and y,and

-   -   S(x,y) is a corresponding element in the similarity matrix.

Finally, cluster analysis converts the distance matrix into treediagrams for analysis.

While this type of program has been very helpful in speeding up theanalysis of card-sorting applications, a problem exists whenparticipants are not familiar with the content of every card. This canhappen, for example, when a company provides a variety of specialized,technical products, such as those shown in FIGS. 1A and 1B. A person whoregularly utilizes some of the products may have little or no knowledgein other products. This type of program has previously required eachparticipant to group every card that was presented to them, regardlessof their knowledge of the content of the card. By forcing the sorting of“unknown” cards, the relationships involving them are skewed. It wouldbe desirable to have a program that did not force such a choice, butthat could deal with this lack of input in some areas.

SUMMARY OF THE INVENTION

The present invention provides a method and computer algorithm forhandling cards that are not sorted by one or more participants and forweighting relational distances accordingly. When a participant does notsort one or more cards, a screen prompt checks to be sure that this isintentional. Then, a record is kept of that card, as well as of thegroups the participant formed. When the matrix of responses is createdfor this participant, any pair that contains an un-selected card isassigned a score of 0. This will be added to the other raw scores toform the summed raw score. At the same time, an unknown matrix isgenerated for the participant. This matrix is initially all zeros.Whenever a participant does not sort one or both of the cards in a pair,the value for that pair is set to one. A total unknown matrix contains asummary of all unknowns for all participants. When the summed raw scoresare normalized, rather than dividing by 2·n, each individual element isdivided by 2·(n−U(x,y)), where n is the total number of participants inthe card-sorting activity and U(x,y) is the number of participants whodid not group at least one of the items in the pair containing x and y.The rest of the calculations remain the same. A final change is made inthe display portion of the program. When the tree structure isdisplayed, any item which some participants did not sort will have afraction shown next to it, giving the number of participants out of thetotal who sorted that item.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features believed characteristic of the invention are setforth in the appended claims. The invention itself, however, as well asa preferred mode of use, further objectives and advantages thereof, willbest be understood by reference to the following detailed description ofan illustrative embodiment when read in conjunction with theaccompanying drawings, wherein:

FIGS. 1A and 1B show exemplary screens during a the execution of firstand second steps of EZSort.

FIG. 2 shows a personal computer.

FIG. 3 shows a block diagram of a computer system in which the disclosedinvention can be used.

FIGS. 4A and 4B is a flowchart representation of a process of cardsorting according to a preferred embodiment of the present invention.

FIGS. 5A-D are exemplary screens for a first user as he works throughthe sorting process.

FIGS. 6A and 6B are exemplary raw score matrices for an individualparticipant and for all participants respectively;

FIGS. 7A and 7B are exemplary unknown matrices for an individualparticipant and for all participants respectively;

FIG. 8 is an exemplary similarity matrix.

FIG. 9 is an exemplary distance matrix;

FIG. 10 is an exemplary tree structure derived from the distance matrixof FIG. 9.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

With reference now to the figures and in particular with reference toFIG. 2, a pictorial representation of a data processing system in whichthe present invention may be implemented is depicted in accordance witha preferred embodiment of the present invention. A computer 200 isdepicted which includes system unit 202, video display terminal 204,keyboard 206, storage devices 208, which may include floppy drives andother types of permanent and removable storage media, and mouse 210.Additional input devices may be included with personal computer 200,such as, for example, a joystick, touchpad, touch screen, trackball,microphone, and the like. Computer 200 can be implemented using anysuitable computer, such as an IBM eServer computer or IntelliStationcomputer, which are products of International Business MachinesCorporation, located in Armonk, N.Y. Although the depictedrepresentation shows a computer, other embodiments of the presentinvention may be implemented in other types of data processing systems,such as a network computer. Computer 200 also preferably includes agraphical user interface (GUI) that may be implemented by means ofsystems software residing in computer readable media in operation withincomputer 200.

With reference now to FIG. 3, a block diagram of a data processingsystem is shown in which the present invention may be implemented. Dataprocessing system 300 is an example of a computer, such as computer 200in FIG. 2, in which code or instructions implementing the processes ofthe present invention may be located. Data processing system 300 employsa peripheral component interconnect (PCI) local bus architecture.Although the depicted example employs a PCI bus, other bus architecturessuch as Accelerated Graphics Port (AGP) and Industry StandardArchitecture (ISA) may be used. Processor 302 and main memory 304 areconnected to PCI local bus 306 through PCI bridge 308. PCI bridge 308also may include an integrated memory controller and cache memory forprocessor 302. Additional connections to PCI local bus 306 may be madethrough direct component interconnection or through add-in boards. Inthe depicted example, local area network (LAN) adapter 310, smallcomputer system interface SCSI host bus adapter 312, and expansion businterface 314 are connected to PCI local bus 306 by direct componentconnection. In contrast, audio adapter 316, graphics adapter 318, andaudio/video adapter 319 are connected to PCI local bus 306 by add-inboards inserted into expansion slots. Expansion bus interface 314provides a connection for a keyboard and mouse adapter 320, modem 322,and additional memory 324. SCSI host bus adapter 312 provides aconnection for hard disk drive 326, tape drive 328, and CD-ROM drive330. Typical PCI local bus implementations will support three or fourPCI expansion slots or add-in connectors.

An operating system runs on processor 302 and is used to coordinate andprovide control of various components within data processing system 300in FIG. 3. The operating system may be a commercially availableoperating system such as Windows XP, which is available from MicrosoftCorporation. An object oriented programming system such as Java may runin conjunction with the operating system and provides calls to theoperating system from Java programs or applications executing on dataprocessing system 300. “Java” is a trademark of Sun Microsystems, Inc.Instructions for the operating system, the object-oriented programmingsystem, and applications or programs are located on storage devices,such as hard disk drive 326, and may be loaded into main memory 304 forexecution by processor 302.

Those of ordinary skill in the art will appreciate that the hardware inFIG. 3 may vary depending on the implementation. Other internal hardwareor peripheral devices, such as flash read-only memory (ROM), equivalentnonvolatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIG. 3. Also, theprocesses of the present invention may be applied to a multiprocessordata processing system.

For example, data processing system 300, if optionally configured as anetwork computer, may not include SCSI host bus adapter 312, hard diskdrive 326, tape drive 328, and CD-ROM 330. In that case, the computer,to be properly called a client computer, includes some type of networkcommunication interface, such as LAN adapter 310, modem 322, or thelike. As another example, data processing system 300 may be astand-alone system configured to be bootable without relying on sometype of network communication interface, whether or not data processingsystem 300 comprises some type of network communication interface. As afurther example, data processing system 300 may be a personal digitalassistant (PDA), which is configured with ROM and/or flash ROM toprovide non-volatile memory for storing operating system files and/oruser-generated data.

The depicted example in FIG. 3 and above-described examples are notmeant to imply architectural limitations. For example, data processingsystem 300 also may be a notebook computer or hand held computer inaddition to taking the form of a PDA. Data processing system 300 alsomay be a kiosk or a Web appliance. The processes of the presentinvention are performed by processor 302 using computer implementedinstructions, which may be located in a memory such as, for example,main memory 304, memory 324, or in one or more peripheral devices326-330.

We will now walk through an embodiment of the innovative process inorder to explain it more fully, starting with reference to FIGS. 4A andB, which show a flowchart of the process and to FIGS. 5-9, which showchoices faced by the user and the matrices created by the program as itworks. The process begins in the same manner as its predecessor—with thepresentation of a screen that offers items for the participant to sort,shown in FIG. 5A. For the sake of simplicity, we will discuss onlyeleven cards in this example, although these may be part of a largerstudy that includes more cards. The eleven cards of interest are shownon the source side of the screen and have arbitrarily been labeled A-Kfor reference. The first user performs his sorting (step 405), as shownin FIG. 5B. This participant has left items G and K unsorted, as he isunfamiliar with these items. He clicks the arrow to indicate that he isthrough sorting (step 410). The program checks to see if any itemsremain in the Source field (step 415); if not, it skips ahead to thenext part of the algorithm (step 430); otherwise, the program visuallymarks (step 420) the remaining items in the Source field and presents(step 425) the screen seen in FIG. 5C. This screen notes that items wereleft in the source side of the screen and seeks to discover if this wasintentional. If it was inadvertent, the user clicks on the “no” buttonand is given another chance to finish sorting (step 415); if the userleft items because he was not familiar with them, the user clicks on the“yes” button and the program proceeds. At the same time, the programsaves a copy of items that were left on the source side as “unknown”. Inthe next part of the input, the previous groups are presented to allowfurther, higher level groupings, if desired. In FIG. 5D, the participanthas made further entries. Note that this user has further grouped onlytwo groups (step 430), the group containing B and I and the groupcontaining only D. These two groups are separated only by a single line,showing that they are grouped together at a higher level, but not at alower level. All other groups are separated by double lines. The inputphase concludes with part 3, not specifically shown, in which the usernames the higher level groups that he has created (step 435). Data willbe gathered from a number of participants, each following the processoutlined above. Once the data is collected, it is analyzed.

The flow for analysis of the data is shown in FIG. 4B. First, a rawscore matrix is formed for each participant (step 450). The raw scorematrix for the first user is shown in FIG. 6A. Comparing this matrix tothe groupings seen in FIGS. 5B and 5C, we can note that items B and Iwere grouped together at both levels and the matrix M(B,I)=2. Likewise,the matrix entries for M(C,H), M(C,J), M(H,J), and M(E,F) are equal to2. Items in the group containing B and I were grouped with items in thegroup containing D at the higher levels, although not at the lowerlevels. Therefore M(B,D) and M(I,D) have values of 1. All other valuesare zero. Note particularly that there are zeros for any pair whichcontains items G or K, which were not sorted at all.

Additionally, an unknown matrix is created (step 455) for the firstuser, shown in FIG. 7A. In this unknown matrix, there is a value of 1for those pairs in which one or both of the items were not sorted; allother values are 0. Thus, any pair containing items G or K is 1.Notably, although there were only two items not sorted, there arenineteen pairs that are affected. This provides some indication of howmuch a mistaken grouping, done because the user wasn't familiar with anitem, can affect an analysis.

In the next step, a total raw score is created for all participants byadding all the values for corresponding matrix positions for allparticipants (step 460). For our hypothetical example, twentyparticipants completed the sorting exercise, with the total raw scoreshown in FIG. 6B. Of these participants, including the firstparticipant, two persons did not sort item G, one did not sort item E,one did not sort item H and one did not sort item K. A total unknownmatrix is formed by adding all the corresponding values from the unknownmatrices for all participants (step 465). FIG. 7B is the total unknownmatrix. This matrix shows how many persons did not address a particularpair.

Next, each of the raw scores is normalized to a value representative ofthe similarity of the items as seen by the participants. This similaritymatrix, shown in FIG. 8, is formed by dividing each total raw score bythe highest score possible for that pair (step 470). Since the number ofpersons NOT answering each question is shown in the total unknownmatrix, the highest score possible for pair x,y is 2·(20−N(x,y)), whereN(x,y) is the corresponding entry in the total unknown matrix and 20 isthe number of participants. To generalize,S(x,y)=R(x,y)/(2·(n−N(x,y)))where S(x,y) is an entry in the similarity matrix for pair x,y;

R(x,y) is a corresponding entry in the total raw score matrix, and

n is the number of participants in the study.

Once the similarity matrix has been created, it is transformed into thedistance matrix (step 480) by subtracting each similarity entry from 1to create the corresponding distance entry. That isD(x,y)=1−S(x,y)Where S(x,y) is an entry in the similarity matrix for pair x and y andD(x,y) is the corresponding entry in the distance matrix. The distancematrix is shown in FIG. 9.

The distance matrix is used to create the tree structure that is outputby the program. FIG. 10 shows that portion of the tree that includes theitems in our example.

As shown by this example, programs such as EZSort are now able toprovide more appropriate relationship information, due to elimination ofthe distortion produced when a user does not understand an entry.

It is important to note that while the present invention has beendescribed in the context of a method run on a computer, those ofordinary skill in the art will appreciate that the processes of thepresent invention are capable of being distributed in the form of acomputer readable medium of instructions and a variety of forms and thatthe present invention applies equally regardless of the particular typeof signal bearing media actually used to carry out the distribution.Examples of computer readable media include recordable-type media, suchas a floppy disk, a hard disk drive, a RAM, CD-ROMs, DVD-ROMs, andtransmission-type media, such as digital and analog communicationslinks, wired or wireless communications links using transmission forms,such as, for example, radio frequency and light wave transmissions. Thecomputer readable media may take the form of coded formats that aredecoded for actual use in a particular data processing system.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

For example, the specific algorithm used here to measure a logical“distance” between items is based on an answer having a scale of 0 to 1.However, any reasonable scale could used, as long as it arrives atrelative distances apart and a different algorithm could be used if itmade allowances for removing items not sorted.

1. A method operative in a data processing system, the method comprisingperforming the following steps for each participant of a plurality ofparticipants: displaying a plurality of item cards, acceptingparticipant input regarding logical groupings into which item cards ofsaid plurality of item cards are sorted, identifying item cards of saidplurality of item cards that are not sorted by the participant into saidlogical groupings, and saving a record of said logical groupings and arecord of said item cards of said plurality of item cards that are notsorted into said logical groupings; and calculating, for all of saidplurality of participants by means of cluster analysis, an indication ofperceived distance between ones of said item cards, wherein saidcalculating step adjusts calculations using said item cards of saidplurality of item cards that are not sorted into said logical groupings.2. The method of claim 1, further comprising, as a step performed foreach participant of said plurality of participants, asking saidparticipant to verify that said participant is not familiar with contentof item cards of said plurality of item cards that are not sorted intosaid logical groupings.
 3. The method of claim 1, further comprising, asa step performed for each participant of said plurality of participants,accepting input regarding higher-level groupings for said item cards ofsaid plurality of item cards that are sorted into said logicalgroupings.
 4. The method of claim 3, further comprising, as a stepperformed for each participant of said plurality of participants,accepting names for higher-level groupings for said item cards of saidplurality of item cards that are sorted into said logical groupings. 5.The method of claim 1, wherein, in said saving step, a raw score matrixis created for each participant with a value for each possible pair ofitems, said value being: 0 if said participant did not group said pairof items together at any level, 1 if said participant grouped said pairof items together at a first level but not at a second level, and 2 ifsaid participant grouped said pair of items together at both said firstlevel and said second level.
 6. The method of claim 5, wherein in saidcalculating step, a total raw score (TRS) matrix is created by summingcorresponding values of said raw score matrix for each participant. 7.The method of claim 1, wherein in said saving step, an unknown matrix iscreated for each participant with a value for each possible pair ofitems, said value being 0 if both of said pair of items are sorted intoone of said logical groupings and 1 if one or both of said pair of itemsare not sorted.
 8. The method of claim 7, wherein in said calculatingstep, a total unknown (TU) matrix is created by summing correspondingvalues of said unknown matrix for each participant.
 9. The method ofclaim 1, wherein in said calculating step, normalized values for eachpair of item cards are calculated by dividing a raw score for a pair ofitem cards by a value equal to a total number of participants respondingless the number of participants who did not group one of said pair ofitem cards.
 10. The method of claim 9, further comprising, in saidcalculating step, subtracting said normalized values from 1 to find acorresponding distance value.
 11. A computer program product in acomputer-readable medium comprising: input instructions to be performedby each participant of a plurality of participants: first instructionsfor presenting a plurality of item cards to be sorted into logicalgroupings; second instructions for accepting participant input thatsorts item cards of said plurality of item cards into logical groupings;third instructions for identifying item cards of said plurality of itemcards that are not sorted by said participant into said logicalgroupings; and fourth instructions for saving information regarding saidlogical groupings and information regarding said item cards of saidplurality of item cards that are not sorted into said logical groupings;and calculating instructions for calculating relative distances betweenones of said item cards, wherein said information regarding said itemcards of said plurality of item cards that are not sorted are used insaid calculating instructions wherein said plurality of instructions arecomputer executable code.
 12. The computer program product of claim 11,further comprising, as part of said input instructions: fifthinstructions for asking said participant to verify that said participantis not familiar with content of item cards of said plurality of itemcards that are not sorted into said logical groupings.
 13. The computerprogram product of claim 11, further comprising, as part of said inputinstructions, fifth instructions for accepting input regardinghigher-level groupings for said item cards of said plurality of itemcards that are sorted into said logical groupings.
 14. The computerprogram product of claim 13, farther comprising, as part of said inputinstructions, sixth instructions for accepting names for higher-levelgroupings for said item cards of said plurality of item cards that aresorted into said logical groupings.
 15. The computer program product ofclaim 11, wherein said fourth instructions create a raw score matrix foreach participant with a value for each possible pair of items, saidvalue being: 0 if said individual participant did not group said twoitems together at any level, 1 if said individual participant groupedsaid two items together at a first level but not at a second level, and2 if said individual participant grouped said two items together at bothsaid first level and said second level.
 16. The computer program productof claim 15, wherein said calculating instructions calculate a total rawscore matrix by summing corresponding values of said raw score matrixfor each participant.
 17. The computer program product of claim 11,wherein said fourth instructions create an unknown matrix for eachparticipant with a value for each possible pair of items, said valuebeing 0 if both of said pair of items are sorted into one of saidlogical groupings and 1 if one or both of said pair of items are notsorted.
 18. The computer program product of claim 17, wherein saidcalculating instructions calculate a total unknown matrix by summingcorresponding values of said unknown matrix for each participant. 19.The computer program product of claim 11, wherein said calculatinginstructions calculate normalized values for each pair of item cards bydividing a raw score for a pair of item cards by a value equal to atotal number of individual participants responding less the number ofparticipants who did not group one of said pair of item cards.
 20. Thecomputer program product of claim 11, wherein said calculatinginstructions comprise subtracting a normalized value from 1 to obtain adistance value.
 21. A method operative in a data processing system, themethod comprising: performing the following steps for each participantof a plurality of participants: displaying a plurality of item cards,accepting participant input regarding logical groupings into which itemcards of said plurality of item cards are sorted, identifying item cardsof said plurality of item cards that are not sorted by the participant;and asking said participant of said plurality of participants to verifythat said participant is not familiar with content of item cards of saidplurality of item cards that are not sorted, and calculating a raw scorevalue for each pair of items according to a value of: 0 if saidparticipant did not group said pair of items together at any level, 1 ifsaid participant grouped said pair of items together at a first levelbut not at a second level, and 2 if said participant grouped said pairof items together at both said first level and said second level,calculating an unknown value for each pair of items according to thevalue of: 0 if both of said pair of items are sorted into one of saidlogical groupings, and 1 if one or both of said pair of items are notsorted after said performing step, calculating, for each pair of items iand j, the following: a total raw score R(i,j), determined by summingcorresponding values from said raw score value for each of saidparticipants, a total unknown score U(i,j), determined by summingcorresponding values from said unknown value for each participant, asimilarity calculation according to the formulaS(i,j)=R(i,j)/(n−U(i,j)), where n is the number of participants, and adistance calculation according to the formula D(i,j)=1−S(i,j).
 22. Acomputer system comprising: receiving means for receiving input; outputmeans for delivering output; a processor, connected to said receivingmeans and to said output means, to process information; storage,connected to said processor, in which to store information; andinstructions, stored in said storage for execution by said processor,said instructions comprising: input instructions to be performed by eachparticipant of a plurality of participants; first instructions forpresenting a plurality of item cards to be sorted into logicalgroupings; second instructions for accepting participant input thatsorts item cards of said plurality of item cards into said logicalgroupings; third instructions for identifying item cards of saidplurality of item cards that are not sorted by the participant into saidlogical groupings; fourth instructions for saving information regardingsaid logical groupings and item cards of said plurality of item cardsthat are not sorted into said logical groupings; and calculatinginstructions for calculating relative distances between ones of saiditem cards, wherein said plurality of instructions are computerexecutable code, and wherein said information regarding said item cardsof said plurality of item cards that are not sorted are used in saidcalculating instructions.
 23. The computer system of claim 22, furthercomprising, as part of said input instructions, asking said participantto verify that said participant is not familiar with content of saiditem cards of said plurality of item cards that are not sorted.
 24. Thecomputer system of claim 22, further comprising, as part of said inputinstructions, fifth instructions for accepting input regardinghigher-level groupings for said item cards of said plurality of itemcards that are sorted.
 25. The computer system of claim 24, furthercomprising, as part of said input instructions, sixth instructions foraccepting names for higher-level groupings for said item cards of saidplurality of item cards that are sorted.
 26. The computer system ofclaim 22, wherein said fourth instructions create a raw score matrix foreach participant with a value for each possible pair of items, saidvalue being: 0 if said individual participant did not group said twoitems together at any level, 1 if said individual participant groupedsaid two items together at a first level but not at a second level, and2 if said individual participant grouped said two items together at bothsaid first level and said second level.
 27. The computer system of claim26, wherein said calculating instructions calculate a total raw scorematrix by summing corresponding values of said raw score matrix for eachparticipant.
 28. The computer system of claim 22, wherein said fourthinstructions create an unknown matrix for each participant with a valuefor each possible pair of items, said value being 0 if both of said pairof items are sorted into one of said logical groupings and 1 if one orboth of said pair of items are not sorted.
 29. The computer system ofclaim 28, wherein said calculating instructions calculate a totalunknown matrix by summing corresponding values of said unknown matrixfor each participant.
 30. The computer system of claim 22, wherein saidcalculating instructions calculate normalized values for each pair ofitem cards by dividing a raw score for a pair of item cards by a valueequal to a total number of individual participants responding less thenumber of participants who did not group one of said pair of item cards.31. The computer system of claim 22, wherein said calculatinginstructions comprise subtracting a normalized value from 1 to obtain adistance value.